feat: project setup and upload file #1712

0marperez · 2025-10-17T13:55:48Z

Issue #

N/A

Description of changes

Project setup
Publishing config
Transfer manager client
Business metric
Transfer interceptors
Upload file operation
- Concurrent uploads
- MPU part buffering
- Code generated IO
- Code generated type converters

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

github-actions · 2025-10-17T14:02:55Z

A new generated diff is ready to view.

No codegen difference in the AWS SDK

github-actions · 2025-10-17T14:14:53Z

A new generated diff is ready to view.

No codegen difference in the AWS SDK

github-actions · 2025-10-17T14:42:33Z

A new generated diff is ready to view.

No codegen difference in the AWS SDK

github-actions · 2025-10-17T14:53:02Z

A new generated diff is ready to view.

No codegen difference in the AWS SDK

github-actions · 2025-10-17T15:11:43Z

A new generated diff is ready to view.

No codegen difference in the AWS SDK

lauzadis

Nice start

...mmon/src/aws/sdk/kotlin/runtime/http/interceptors/businessmetrics/AwsBusinessMetricsUtils.kt

hll/build.gradle.kts

hll/s3-transfer-manager/build.gradle.kts

hll/s3-transfer-manager/jvm/test/aws/sdk/kotlin/hll/s3transfermanager/UploadFileTest.kt

hll/s3-transfer-manager/common/src/aws/sdk/kotlin/hll/s3transfermanager/utils/UploadFile.kt

...ransfer-manager/common/src/aws/sdk/kotlin/hll/s3transfermanager/BusinessMetricInterceptor.kt

hll/s3-transfer-manager/common/src/aws/sdk/kotlin/hll/s3transfermanager/S3TransferManager.kt

...-transfer-manager/common/src/aws/sdk/kotlin/hll/s3transfermanager/model/UploadFileRequest.kt

hll/s3-transfer-manager/common/src/aws/sdk/kotlin/hll/s3transfermanager/utils/UploadFile.kt

hll/s3-transfer-manager/common/src/aws/sdk/kotlin/hll/s3transfermanager/S3TransferManager.kt

hll/s3-transfer-manager/common/src/aws/sdk/kotlin/hll/s3transfermanager/TransferInterceptor.kt

hll/s3-transfer-manager/build.gradle.kts

github-actions · 2025-10-21T14:07:26Z

A new generated diff is ready to view.

No codegen difference in the AWS SDK

github-actions · 2025-10-30T03:03:52Z

A new generated diff is ready to view.

AWS SDK (ignoring whitespace)

github-actions · 2025-10-30T04:09:45Z

A new generated diff is ready to view.

AWS SDK (ignoring whitespace)

github-actions · 2025-10-30T15:51:50Z

A new generated diff is ready to view.

AWS SDK (ignoring whitespace)

github-actions · 2025-10-30T16:13:52Z

A new generated diff is ready to view.

AWS SDK (ignoring whitespace)

gradle/libs.versions.toml

...n/kotlin/aws/sdk/kotlin/hll/dynamodbmapper/codegen/operations/rendering/OperationRenderer.kt

hll/s3-transfer-manager-codegen/build.gradle.kts

...odegen/src/main/kotlin/aws/sdk/kotlin/hll/s3transfermanager/codegen/mappings/MappingTypes.kt

hll/s3-transfer-manager/build.gradle.kts

hll/s3-transfer-manager/common/src/aws/sdk/kotlin/hll/s3transfermanager/S3TransferManager.kt

...-manager/common/src/aws/sdk/kotlin/hll/s3transfermanager/operations/uploadfile/UploadFile.kt

...ger/common/src/aws/sdk/kotlin/hll/s3transfermanager/operations/uploadfile/HelperFunctions.kt

...mon/src/aws/sdk/kotlin/hll/s3transfermanager/operations/uploadfile/hooks/InitiateTransfer.kt

...common/test/aws/sdk/kotlin/hll/s3transfermanager/operations/uploadobject/UploadObjectTest.kt

github-actions · 2025-10-30T20:40:09Z

A new generated diff is ready to view.

AWS SDK (ignoring whitespace)

hll/s3-transfer-manager-codegen/build.gradle.kts

settings.gradle.kts

gradle/libs.versions.toml

ianbotsf · 2025-10-31T22:58:52Z

hll/build.gradle.kts

+
+    "s3-transfer-manager-codegen", // TODO: Disable publishing ?


Comment: 👍 Yes, this is in the right place. We don't want to publish this since we have no use case for it right now. We can scratch the TODO.

This doesn't disable publication right? Just API validation and docgen

This is configured in repo tools right?

ianbotsf · 2025-10-31T22:59:45Z

hll/hll-codegen/build.gradle.kts

+
+java {
+    sourceCompatibility = JavaVersion.VERSION_1_8
+    targetCompatibility = JavaVersion.VERSION_1_8
+}
+
+tasks.withType<KotlinCompile> {
+    compilerOptions {
+        jvmTarget.set(JvmTarget.JVM_1_8)
+    }
+}


Question: Why are these additions necessary?

I was getting error messages about this being compiled with a higher version of Java than S3 TM codegen (11 vs 8)

ianbotsf · 2025-11-03T18:16:46Z

...-manager/common/src/aws/sdk/kotlin/hll/s3transfermanager/operations/uploadfile/UploadFile.kt

+    val mpuUploadId = initiateTransfer(
+        multipartUpload,
+        transferContext,
+        contentLength,
+        uploadFileRequest,
+        interceptors,
+        client,
+    )
+
+    val uploadedParts = transferBytes(
+        multipartUpload,
+        contentLength,
+        partSizeBytes,
+        logger,
+        uploadFileRequest,
+        transferContext,
+        mpuUploadId,
+        interceptors,
+        client,
+        maxInMemoryParts,
+        maxConcurrentPartUploads,
+    )
+
+    completeTransfer(
+        multipartUpload,
+        transferContext,
+        uploadFileRequest,
+        mpuUploadId,
+        uploadedParts,
+        interceptors,
+        client,
+    )


Style: The volume of arguments passed into these functions is too great. Can any of these be grouped into objects, derived from other parameters, etc.? This level of data coupling is an indicator that we might be better served modelling a base operation type which can be implemented for each operation type (e.g., UploadFile) or for each subtype (e.g., UploadFileSingle and UploadFileMultipart), which would reduce the amount of if (multipartUpload) calls.

I left a TODO for this, once we write other operations it'll be easier to create a common abstraction.

...c/main/kotlin/aws/sdk/kotlin/hll/s3transfermanager/codegen/mappings/uploadfile/Converters.kt

...common/src/aws/sdk/kotlin/hll/s3transfermanager/operations/uploadfile/hooks/TransferBytes.kt

ianbotsf · 2025-11-03T18:32:36Z

...mon/src/aws/sdk/kotlin/hll/s3transfermanager/operations/uploadobject/phases/TransferBytes.kt

+) = produce(
+    capacity = maxInMemoryParts,
+) {


Correctness: Isn't maxInMemoryParts supposed to limit the parts for the entire S3TM? This looks like it applies to individual objects but we'll be parallelizing multi-object transfers.

I decided to use a Channel for each object but while keeping track of total parts in memory using a Semaphore.

...common/src/aws/sdk/kotlin/hll/s3transfermanager/operations/uploadfile/hooks/TransferBytes.kt

lauzadis · 2025-11-03T19:13:26Z

hll/build.gradle.kts

+
+    "s3-transfer-manager-codegen", // TODO: Disable publishing ?


This doesn't disable publication right? Just API validation and docgen

lauzadis · 2025-11-03T19:19:16Z

...c/main/kotlin/aws/sdk/kotlin/hll/s3transfermanager/codegen/mappings/uploadfile/Converters.kt

+internal val uploadFileConversions = listOf(
+    ConversionMapping(
+        source = TypeRef(
+            "aws.sdk.kotlin.services.s3.model",
+            "PutObjectResponse",
+        ),
+        destination = TypeRef(
+            "aws.sdk.kotlin.hll.s3transfermanager.model",
+            "UploadFileResponse",
+        ),
+        setOf(
+            "bucketKeyEnabled",
+            "checksumCrc32",
+            "checksumCrc32C",
+            "checksumCrc64Nvme",
+            "checksumSha1",
+            "checksumSha256",
+            "checksumType",
+            "eTag",
+            "expiration",


I don't think a data string will make this any more readable / maintainable? The internal spec already has this modeled as a JSON list, we can make changes (which I believe are unlikely) by inspecting the diff of that file

hll/s3-transfer-manager/build.gradle.kts

lauzadis · 2025-11-03T19:27:25Z

hll/s3-transfer-manager/common/src/aws/sdk/kotlin/hll/s3transfermanager/S3TransferManager.kt

+    /**
+     * Preferred part size for multipart uploads.
+     * If using this size would require more than 10,000 parts (the S3 limit),
+     * the smallest possible part size that results in 10,000 parts is used instead.
+     *
+     * Default to 8,000,000 bytes.
+     */
+    public val partSizeBytes: Long = builder.partSizeBytes


This is my fault since I left a comment asking to change it to just partSize to simplify the name. We are logging a warning when deviating from the configured part size. It's not a strong opinion so I will let @0marperez make the decision

#1712

github-actions · 2025-11-07T00:13:18Z

A new generated diff is ready to view.

AWS SDK (ignoring whitespace)

0marperez added 6 commits October 6, 2025 17:39

feat: s3 transfer manager client

bc2fa0a

saving work - upload file

6895fc3

self review

c9486f1

pull from main

22243e7

misc: merge from main

dbff057

finish upload file

835866a

provide static dummy credentials to tests

8edf604

0marperez added the no-changelog Indicates that a changelog entry isn't required for a pull request. Use sparingly. label Oct 17, 2025

add s3 as a test dependency?

4137873

refactor tests to be JVM only

d84516d

Depend on missing 'DefaultAwsSigner'

9d5ec07

0marperez marked this pull request as ready for review October 17, 2025 15:14

0marperez requested a review from a team as a code owner October 17, 2025 15:14

lauzadis requested changes Oct 17, 2025

View reviewed changes

Merge branch 'main' of github.com:aws/aws-sdk-kotlin into s3-tm-dev

71ea5b7

ianbotsf requested changes Oct 20, 2025

View reviewed changes

feedback

fb2417f

0marperez mentioned this pull request Oct 30, 2025

feat: sdk source readRemaining smithy-lang/smithy-kotlin#1454

Open

feedback v2

a193a2f

add missing KDocs and self review

010bd4b

lauzadis mentioned this pull request Oct 30, 2025

S3 Transfer Manager #972

Open

1 task

remove old TODO

8a821d6

latest feedback

ac6c88e

lauzadis reviewed Oct 30, 2025

View reviewed changes

feedback v3

32aae24

ianbotsf requested changes Nov 3, 2025

View reviewed changes

lauzadis approved these changes Nov 3, 2025

View reviewed changes

feedback round 3?

ac24d5a


		"s3-transfer-manager-codegen", // TODO: Disable publishing ?

feat: project setup and upload file #1712

Are you sure you want to change the base?

feat: project setup and upload file #1712

Conversation

0marperez commented Oct 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Issue #

Description of changes

Uh oh!

github-actions bot commented Oct 17, 2025

Uh oh!

github-actions bot commented Oct 17, 2025

Uh oh!

github-actions bot commented Oct 17, 2025

Uh oh!

github-actions bot commented Oct 17, 2025

Uh oh!

github-actions bot commented Oct 17, 2025

Uh oh!

lauzadis left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Oct 21, 2025

Uh oh!

github-actions bot commented Oct 30, 2025

Uh oh!

github-actions bot commented Oct 30, 2025

Uh oh!

github-actions bot commented Oct 30, 2025

Uh oh!

github-actions bot commented Oct 30, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Oct 30, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

0marperez commented Oct 17, 2025 •

edited

Loading